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Abstract. We study the verification of a finite continuous-time Markov chain (CTMC) 
C against a linear real-time specification given as a deterministic timed automaton (DTA) 
A with finite or MuUer acceptance conditions. The central question that we address 
is: what is the probability of the set of paths of C that are accepted by A, i.e., the 
likelihood that C satisfies A? It is shown that under finite acceptance criteria this equals 
the reachability probability in a finite piecewise deterministic Markov process (PDF), 
whereas for MuUer acceptance criteria it coincides with the reachability probability of 
terminal strongly connected components in such a FDP. Qualitative verification is shown 
to amount to a graph analysis of the PDF. Reachability probabilities in our FDPs are then 
characterized as the least solution of a system of Volterra integral equations of the second 
type and are shown to be approximated by the solution of a system of partial differential 
equations. For single-clock DTA, this integral equation system can be transformed into 
a system of linear equations where the coefficients are solutions of ordinary differential 
equations. As the coefficients are in fact transient probabilities in CTMCs, this result 
implies that standard algorithms for CTMC analysis suffice to verify single-clock DTA 
specifications. 
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1. Introduction 

Continuous-time Markov chains (CTMCs) are one of the most prominent models in 
performance and dependabihty analysis. They are exploited in a broad range of applications, 
and constitute the underlying semantical model of a plethora of modeling formalisms for 
real-time probabilistic systems such as Markovian queueing networks, stochastic Petri nets, 
stochastic variants of process algebras, and calculi for systems biology. CTMC model 
checking has been mainly focused on the branching-time temporal logic CSL (Continuous 
Stochastic Logic [3l [7]), a variant of timed CTL where the CTL universal and existential 
path quantifiers are replaced by a probabilistic operator. Like CTL model checking, CSL 
model checking of finite CTMCs proceeds by a recursive descent over the parse tree of the 
CSL formula. One of the key ingredients is that time-bounded reachability probabilities can 
be approximated arbitrarily closely by a reduction to transient analysis in CTMCs This 
results in an efficient polynomial-time algorithm that has been realized in model-checking 
tools such as PRISM [19] and MRMC [20] and has been successfully applied to various case 
studies from diverse application areas. 

Verifying a finite CTMC C against linear-time (but untimed) specifications in the form 
of a regular or w-regular language is rather straightforward and boils down to computing 
reachability probabilities in discrete-time Markov chains (DTMCs). This can be seen as 
follows. Assume that the specification is provided as a deterministic automaton A on finite 
words, or alternatively as a deterministic Muller automaton A. The underlying idea is that 
the evolution of a CTMC is "synchronized" with an accepting run of A by considering the 
state labels in a CTMC, i.e., atomic propositions, as letters read by A. As A does not 
constrain the timing of events in the CTMC C, it suffices to take a synchronous product 
of A and Cs embedded DTMC, denoted emb(C), which is obtained by just ignoring the 
random state residence times in C while keeping all other ingredients, in particular the 
transition probabilities and state labels. For finite acceptance criteria, the probability that 
C \= A, i.e., the probability of the set of paths in C that are accepted by A, Pr(C \= A) 
for short, is obtained as the reachability probability in the product emb(C) <^ A of the 
final states in A. Since A is deterministic, emb{C) ^ A is a DTMC. In case of Muller 
acceptance criteria, Pr(C \= A) corresponds to the reachability probability of accepting 
terminal strongly connected components in enib{C) ^ A. This follows directly from results 
in [13]. The reachability probabilities in a DTMC can be obtained by solving a system of 
linear equations whose size is linear in the size of the DTMC, see, e.g., jl8]. 

In this paper, we consider the verification of CTMCs against linear real-time specifica- 
tions that are given as deterministic timed automata (DTA) [1]. That is to say, we explore 
the following problem: given a CTMC C, and a linear real-time specification provided as a 
deterministic timed automaton A, what is the probability of the set of paths of C that are 
accepted by A, i.e., what is Pr(C \= A)7 

Example 1.1. Let us illustrate the usage of DTA specifications by means of a small 
example. Consider a robot randomly moving in some area. It starts in some zone {A, say) 



and has to reach zone B within 10 time units, cf. Figure 1(a) (For simplicity, all zones on 
the map are equally-sized, but this is not a restriction.) The robot randomly moves through 
the zones, and resides in a zone for an exponentially distributed amount of time. The robot 
may pass through all zones to reach B, but should not stay longer than 2 time units in any 
gray zone. The specification "reach B from A within 10 time units while residing in any 
gray zone for at most 2 time units" is modeled by a simple DTA which accepts once location 
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q2 is reached, cf. Figure 1(b) Clock x controls the timing constraint on the residence times 
of the gray zones (assumed to be labeled with g), while clock y controls the global time 
constraint to reach zone B. In state qq, the robot traverses non-gray zones, in qi gray zones, 
and in q2 it has reached the goal zone B. 




(a) Robot map (b) Two-clock DTA 

Figure 1: A robot example 

Like in the untimed setting discussed before, we consider two variants: DTA that 
accept finite timed words, and DTA that accept infinite timed words according to a Muller 
acceptance condition. (Note that DTA with Muller acceptance condition are strictly more 
expressive than DTA with Biichi acceptance conditions [IJ.) The considered verification 
problem is substantially harder than the case for untimed linear specifications, e.g., as 
the DTA may constrain the timing of events in C, it does not suffice to take the embedded 
DTMC emb(C) as starting-point. In addition, the product of a CTMC and a DTA is neither 
a CTMC nor a DTA, and has an infinite state space. It is unclear which (and whether a) 
stochastic process is obtained from such infinite product, and if so, how to analyze it. 

We tackle the verification of a finite CTMC against a DTA specification as follows: 

(1) We first show that the problem C ^ ^ is well-defined in the sense that the set of paths 
of C that are accepted by A is measurable. 

(2) We define the product C (S) A for CTMC C and DTA ^ as a variant of DTA in 
which, besides the usual ingredients of timed automata like guards and clock resets, the 
location residence time is exponentially distributed, and define a probability space over 
sets of timed paths in this model. In particular, we show that the probability of C |= ^ 
coincides with the reachability probability of accepting paths in C ® A. 

(3) We adapt the standard region construction for timed automata [IJ to this variant of 
DTA, and show that the thus obtained region automata are in fact piecewise determin- 
istic Markov processes (PDPs) [16], a model that is frequently used in, e.g., stochastic 
control theory and financial mathematics. The characterization of region automata as 
PDPs sets the ground for obtaining the following results concerning qualitative and 
quantitative verification of CTMCs against DTA. 

(4) For finite acceptance criteria, we show that Pr(C \= A) equals the reachability proba- 
bility in the embedded PDP of C ® A. Under Muller acceptance criteria, Pr(C |= A) 
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equals the reachability probability of accepting terminal strongly connected components 
in this embedded PDP. In case of qualitative verification — does CTMC C satisfy A with 
probability larger than zero, or equal to one? — a graph traversal of the (embedded) 
PDP suffices. 

(5) We then show that reachability probabilities in our PDPs can be characterized as the 
least solution of a system of Volterra integral equations of the second type [2]. This 
probability is shown to be approximated by the solution of a system of partial differential 
equations (PDEs). 

(6) For the case of single-clock DTA, we show that the system of integral equations can 
be transformed into a system of linear equations, whose coefficients are solutions of 
some ordinary differential equations (ODEs). For these coefficients either an analytical 
solution (for small state space) can be obtained or an arbitrarily closely approximated 
solution can be determined efficiently. 

Related work. Model checking CTMCs against linear real-time specifications has received 
scant attention so far. To our knowledge, this issue has only been (partly) addressed in 
[171 E]- Baier et al. [6] define the logic asCSL where path properties are characterized by 
(time-bounded) regular expressions over actions and state formulas. The truth value of path 
formulas depends not only on the available actions in a given time interval, but also on the 
validity of certain state formulas in intermediate states. asCSL is strictly more expressive 
than CSL [6j. Model checking asCSL is performed by representing the regular expressions 
as finite-state automata, followed by computing time-bounded reachability probabilities in 
the product of CTMC C and this automaton. In CSL"""^ |17j . time constraints of until 
modalities are specified by single-clock DTA; the resulting logic is at least as expressive as 
asCSL [T7]. The combined behavior of C and DTA A is interpreted as a Markov renewal 
process and model checking CSL"*"^ is reduced to computing reachability probabilities in 
a DTMC whose transition probabilities are given by subordinate CTMCs. This paper 
takes a completely different approach. The technique of |17j cannot be generalized to 
multiple clocks, whereas our approach does not restrict the number of clocks and thus 
supports more specifications than CSL^"^. The DTA specification of our robot example, 
for instance, can neither be expressed in CSL"'"^ nor in asCSL. For the single-clock case, 
our approach produces the same result as [T7], but yields a (in our opinion) conceptually 
simpler formulation whose correctness can be derived by simplifying the system of integral 
equations obtained for the general case. Moreover, measurability has not been addressed in 
[T7] . Other related work [U [5l [10] provides a quantitative interpretation to timed automata 
where delays and discrete choices are interpreted probabilistically. In this approach, delays 
of unbounded clocks are governed by exponential distributions like in CTMCs. Decidability 
results have been obtained for almost-sure properties [5] and quantitative verification [TO] 
for (a subclass of) single-clock timed automata. 

Organization of the paper. Section [2] defines the three models that are central to this 
paper: CTMCs, DTA, and PDPs. Section [3] shows that the set of paths in CTMC C accepted 
by DTA A is measurable and coincides with reachability probabilities in the product C^A. 
It also shows that the underlying region graph of C A is a (simple instance of a) PDP. 
Section |4| constitutes the main part of the paper and deals with the verification of DTA 
with finite acceptance conditions, and analyzes the quantitative reachability problem in our 
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PDPs, for both the general case and single-dock DTA. Section [5] considers DTA with Muller 
acceptance criteria, as well as qualitative verification. Finally, section [6] concludes. 

This paper extends the conference paper [H] with complete proofs, illustrative exam- 
ples, and by considering Muller acceptance criteria. 

2. Preliminaries 

Given a set let Pr : J-{H) — )• [0, 1] be a probability measure on the measurable space 
{H,T{H)), where J~{H) is a cr-algebra over H. Let Distr{H) denote the set of probability 
measures on this measurable space. 

2.1. Continuous-time Markov chains. 

Definition 2.1 (CTMC). A (labeled) continuous-time Markov chain (CTMC) is a tuple 
C = (S", AP, L, a, P, i?) where S" is a finite set of states; AP is a finite set of atomic 
propositions; L : 5* — )• 2"^^ is the labeling function; a € Distr{S) is the initial distribution; 
P : S" X S" — )■ [0, 1] is a stochastic transition probability matrix; and E : S M^o is the exit 
rate function. 

The probability to exit state s in t time units is given by J^E{s)-e-^^'^^dT; the prob- 
ability to take the transition s — >■ s' in t time units equals P(s, ■s')-/q -E'(,s)e~^^'^^"^dr. A 
state s is absorbing if P(s,s) = 1. The embedded discrete-time Markov chain (DTMC) of 
CTMC C is obtained by deleting the exit rate function E, i.e., emb{C) = {S, AP, L, a, P). 

Definition 2.2 (Timed paths). Let C be a CTMC. Paths^ := S x (M>o x S*)" is the set of 
paths of length n in C; the set of finite paths in C is defined by Paths'^ = UneN PC'ths^ and 
Paths'^ := {S X M>o)'^ is the set of infinite paths in C. Paths^ = Paths'^ U Paths'^ denotes 
the set of all paths in C. 

We denote a path p G Paths'^ (sq) (p € Paths (so) for short) as the sequence p = 
So si S2 • • • starting in state sq such that for n ^ \p\ {\p\ is the number of transitions 
in p a p is finite); p[n] := s„ is the n-th state of p and p{n) := tn is the time spent in state 
Sn- Let p@t be the state occupied in p at time t G Mj.o, i.e. p@t := p[n] where n is the 
smallest index such that Y17=o P{^) > assume w.l.o.g. ti > for any i. 

The definition of a Borel space on paths through CTMCs follows [7]. A CTMC C 
yields a probability measure Pr'' on paths as follows. Let sq, . . ., G S* with P(sj, Sj+i) > 
for ^ i < /c and Iq, . . ., /fc-i nonempty intervals in M^o- Let C(so, /q; • • •> -^fc-ij Sk) denote 
the cylinder set consisting of all paths p G Paths {sq) such that p[i] = Si {i ^ k), and 
p{i) & li {i < k). J^{Paths{sQ)) is the smallest cj-algebra on Paths{sQ) which contains all 
sets C{so, If), . . ., Ik-i, Sk) for all state sequences (so,...,sa;) £ S^~^^ with P(sj,Sj+i) > 
{0 ^ i < k) and Iq, . . ., I^-i range over all sequences of nonempty intervals in Mj.o- The 
probability measure Pr'' on T{Paths{so)) is the unique measure defined by induction on k 
by Pr'^(C(so)) = a(so) and for k > 0: 

Pr^{C{so, lo, ■ ■ ., 4-1, Sfc)) = Pr^ (C(so, Iq, ■ ■ ■Jk-2, Sfc-i)) 

• [ F{sk-i,Sk)E{sk-iye-''^"'-'^^dT. (2.1) 
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7-2 {b} 




rz {c} 

Figure 2: An example CTMC 



Example 2.3. An example CTMC is illustrated in Figure[2l where AP = {a, 6, c} and sq is 
the initial state, i.e., a(so) = 1 and a{s) = for any s ^ sq. The exit rates are indicated at 
the states, whereas the transition probabilities are attached to the transitions. An example 
timed path \s p = sq ^'^ > si -""^ > sq si S2 ■ ■ ■ with p[2] = sq and p@6 = p[3] = si. 

2.2. Deterministic timed automata. Let X = {xi, . . be a set of nonnegative real- 
valued variables, called clocks. An ^-valuation is a function r/ : ^ — ?> M^o assigning to each 
variable x a nonnegative real value r]{x). Let V{X) denote the set of all valuations over X. 
A clock constraint on X, denoted by 5, is a conjunction of expressions of the form x txi c for 
clock X € X, comparison operator ix] G {<, ^, >, ^} and c G N. Let CC{X) denote the set of 
clock constraints over X. An .Y-valuation rj satisfies constraint x txi c, denoted rj \= x \xi c, 
if and only if r/(x) ixi c; it satisfies a conjunction of such expressions if and only if rj satisfies 
all of them. Let denote the valuation that assigns to all clocks. For a subset X Q X, 
the reset of X, denoted t][X := 0], is the valuation r/' such that Vx G X. r]'{x) := and 
Vx ^ X. r]'{x) := r]{x). For 5 € M^o and ^Y-valuation r], r]-\-5 is the A'- valuation ij" such that 
Vx E X. r]"{x) := r]{x)+6, which implies that all clocks proceed at the same speed. 

Definition 2.4 (DTA). A deterministic timed automaton (or DTA for short) is a tuple 
A = {T,,X,Q,qQ, Qf,—>) where E is a finite alphabet; X is a finite set of clocks; Q is a 
nonempty, finite set of locations with initial location qo & Q; Qp is the acceptance condition, 
which is either: 

• Qf Q Q, & set of accepting locations (reachability or finite acceptance), or 

• Qt ^ 2*5, an acceptance family (Muller acceptance). 

The relation C Q x T, x CC{X) x 2'^ x Q is the edge relation satisfying: 

(g ""'^'^ > q' and q "'^ > q" with g / g') implies g Ci g' = 0. 

We refer to q "'^'"'^ > q' as an edge, where a £ S is an input symbol, the guard g is a 
clock constraint on the clocks of A, X is the set of clocks that are to be reset and q' is 
the successor location. Intuitively, the edge q > q' asserts that the DTA A can move 

from location q to q' when the input symbol is a and the guard g holds, while the clocks 
in X should be reset when entering q' . DTA are deterministic as they have a single initial 
location, and outgoing edges of a location labeled with the same input symbol are required 
to have disjoint guards. In this way, the next location is uniquely determined for a given 
location and a given clock valuation. In case no guard is satisfied in a location for a given 
clock valuation, time can progress. If the advance of time will never reach a situation in 
which a guard holds, the DTA will stay in that location ad infinitum. Note that DTA do not 
have location invariants, as in safety timed automata. For the sake of simplicity, diagonal 
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{a},x < 1,0 



{b},x > 1,0 

*) y-^ ■ — 



{a}, I < X <2,{x} 

(a) DTA* A 




c, {x} 6, {x} 

(b) DTA" A 



Figure 3: DTA with (a) reachability and (b) Muller acceptance conditions 



y CXI c are not considered. This restriction does, however, not harm the 



constraints hke x 
expressiveness j[9j. 

An (infinite) timed path of DTA A is of the form 9 = qq 
TjQ = 0, and for all j 5^ 0, it holds tj > 0, rjj+tj \= gj, = 



ao,to 



ai,ti 



> • • • such that 
0], where ijj is 



the clock evaluation when entering qj. The definitions on timed paths (such as 9[i\, 0@t, 
and so forth) for CTMCs can readily be adapted for DTA. We consider DTA with two types 
of acceptance criteria. Let DTA^ and DTA"^ denote the set of DTA with reachability and 
Muller acceptance conditions, respectively. DTA denotes the general case covering both 
DTAO and DTA'^. 

Definition 2.5 (DTA accepting paths). An infinite timed path 6 is accepted by a DTA^ 
if 6[i] G Qp for some i ^ 0; 6* is accepted by a DTA*^ if inf{6) € Qj^, where inf{9) is the 
set of states q G Q such that q = qi for infinitely many z ^ 0. 

The timed path 9 is accepted according to a reachability criterion if it reaches some 
final location, whereas it is accepted according to a Muller acceptance condition if the set of 
infinitely visited locations equals some set in Qjr. As a convention, we assume each location 
g G in DTAO to be a sink. 



Example 2.6. Figure 3(a) depicts an example DTA^ over the alphabet {a, 6} with initial 
location go- The timed automaton is deterministic as go is the only initial location and both 
a-labeled edges have disjoint guards. Any timed path ending m. Qp = {qi} is accepting. 

Figure [3 (b) | depicts an example DTA'^ over the alphabet {a, 5, c}. Its initial location is 
go; its Muller acceptance family equals Qjr = {{go, '72}}- Any accepting path should cycle 
between the locations go and gi finitely often, and between go and g2 infinitely often. 

Remark 2.7. [Expressive power of DTA'^] DTA*^ is the set of (deterministic) timed Muller 
automata, (D)MTA, for short. A (deterministic) timed Biichi automaton, (D)TBA for 
short, has a set Qp oi accepting locations, and accepts an infinite timed path 6 \i 6 visits 
some location in Qp infinitely often, i.e., inf (0) r\Qp ^ 0. The expressive power of (D)TMA 
and (D)TBA is related as follows [1]: 

TMA = TBA > DTMA > DTBA. 

Note that in nondeterministic TMA and TBA, guards on edges emanating from a loca- 
tion may overlap. DTMA are closed under all Boolean operators (union, intersection, and 
complement), while DTBA are not closed under complement. 

Remark 2.8. [Successor location] Since DTA are deterministic, the edge relation — )• can be 
replaced by a (partial) function succ : Qx^xCC{X)^2^xQ. If only the successor location 
is of interest, we simpy use the function succ : Q x S x CC{X) H- Q, i.e., q' = succ{q,a,g). 
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2.3. Piecewise-deterministic Markov processes. PDPs [15] constitute a general model 
for stochastic systems without diffusions [16] and has been applied to a variety of problems 
in engineering, operations research, management science, and economics. Powerful analysis 
and control techniques for PDPs have been developed [23^ [2H [13j . A PDP is a hybrid 
stochastic process involving discrete control (i.e., locations) and continuous variables. 

Let us introduce some auxiliary notions. Let X = {xi, . . . be a set of variables 
in M. Note that clock variables are a special case of these variables. A constraint over 
X, denoted by g, is a subset of M". Let B{X) denote the set of constraints over X. An 
Af-valuation rj satisfies constraint g, denoted t] \= g, if and only if (r/(xi), ...,r]{xn)) G g- For 
g G S{X), a constraint over X = {xi, . . . , Xn}, let 5 be the closure of g, g the interior of 
g, and dg = 'g\ g the boundary of g. For instance, for g = x^ — 2x2 ^ 1.5 A X3 > 2, we 
have g = x\ — 2x2 < 1.5 A X3 > 2, 5 = — 2x2 ^ 1-5 A X3 ^ 2, and dg equals x\ — 2x2 = 
1.5 A X3 = 2. 

To each control location z of a PDP, an invariant Inv{z) is associated, a constraint 
over X which constrains the variable values in z. The state of a PDP is a pair (z, rj) with 
control location z and r] a variable valuation. Let § = {{z,ri) \ z Z,7] \= Inv(z) }, where 
Z is the set of locations. The notions of closure, interior and boundary can be lifted to S 
in a straightforward manner, e.g., dS = U^e^i-^i ^ dlnv{z) is the boundary of §; S and S 
are defined in a similar way. 

Definition 2.9 (PDP [16]). A piecewise-deterministic [Markov) process (PDP) is a tuple 
Z = (Z, X , Inv, (j), A, ^) where Z is a finite set of locations, X is a finite set of variables, 
Inv : Z B{X) is an invariant function, and 

• (p : Z X V{X) X M ^ ^{^) is a flow function, which is the solution of a system of ODEs 
with a Lipschitz continuous vector field, 

• A : S — )■ M^o is an exit rate function satisfying for any ,^ G S: 

3e{C) > 0. function t ^ A(^ei) is integrable on [0, e(^)), (A) 

where {z, rf) ®t = (z, 4){z, tj, t)) , and 

• H : S —?■ Distr{§i) is the transition probability function satisfying: 

/i(^, {^}) = and ^ 1-^ ^{S,,A) is measurable for any A G T{E), 

where ^(^, ^) denotes {iJ,{S,)){A), J^{S) is a cr-algebra generated by Uzezi-^} ^ ^2 with 
Az C T{Inv{z)), and T{Inv{z)) is a cr-algebra generated by Inv{z). 

Let us explain the behavior of a PDP. A PDP can reside in a state = (z, ?]) G S as long 
as Inv{z) holds. In state ^ = {z,rj), the PDP can either delay or take a Markovian jump. 
Delaying by t time units yields the next state = (B t, i.e., the PDP remains in location 
z while all its continuous variables are updated according to (j){z,T],t). The flow function cf) 
defines the time-dependent behavior in a single location, in particular, it specifies how the 
variable valuations change when time elapses. In case of a Markovian jump in state ^, the 
next state ^" = (z",rj") G S is reached with probability /i(^, {^"}). The residence time of a 
state is exponentially distributed; this is defined by the function A. A third possibility for 
a PDP to evolve is by taking forced transitions. When the variable valuation rj satisfies the 
boundary of the invariant, i.e., rj \= dlnv{z), the PDP is forced to take a boundary jump, 
i.e., it has to leave state ^. With probability ^(^, {^"}) it then moves to state For any 
T G M^O) the function A is integrable as the interval [0, T] can be divided into finitely many 
small intervals, on which by equation (A), the function A is integrable. 
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Figure 4: An example PDF with constant exit rate 5 and boundary measure 
^((zo,2),{(zi,2)}) :=1 

A FDF is named piecewise-deterministic because in each location (one piece) the be- 
havior is deterministically determined by the flow function (p. The FDF is Markovian as 
the current state contains all the information to determine the future progress of the FDF. 

2.4. Embedded PDP. The embedded discrete-time Markov process (DTMF) emh{Z) of 
the FDF Z has the same state space ^ as Z and is equipped with a transition probability 
function /t. The one-jump transition probability from a state to a set A C S of states 
(with different location as ^), denoted jj,{S,,A), is given by [16j: 

/i(^,A) = / {QlA){C®tyA{C®t)e-^oHi®r)dr (2.2) 
Jo 

+ (QlA)(^eb(0)-e-^o*"^«®")'^" (2.3) 

where b(^) = inf{t > | ^ ® f G d§} is the minimal time to hit the boundary if such time 
exists; b(^) = oo otherwise. (Ql^)(^) = Jg 1a(^')/^('^) '^?') is the accumulative (one-jump) 
transition probability from ^ to A and 1^(0 is the characteristic function such that 1^(0 = 
1 when ^ € A and 1a (0 = otherwise. Term (12. 2p specifies the probability to delay to 
state ^ © t (on the same location) and take a Markovian jump from ^ © t to A. Note the 
delay t can take a value from [0,b(.^)). Term ()2.3p is the probability to stay in the same 
location for b(.^) time units and then it is forced to take a boundary jump from ^ © b(^) to 
A since Inv{z) will be by any delay invalid. 

Example 2.10. Figure S] depicts a 3-location FDF Z with X = x, where Inv(zo) = x < 2 
and Inv{zi) = Inv{z2) = x ^ K^o- Solving x = 1 yields the flow function cj}{zi,r]{x),t) = 
r](x)+t for i = 0,1,2. The state space of 2 is S = {{zQ,ri) \ r](x) < 2} U {(zi, Mj>o)} U 
{{z2,^^o)}. Let exit rate A(^) = 5 for any ^ G S. For r/ ^ /n?;(zo), let ^((zq, r/), {(zi , r/)}) := 
^, ^[{zo,r]), {{z2,ri)}) := | and the boundary measure be given as /(/((zq, 2), {{zi, 2)}) := 1. 
The time for = (-^cO) to hit the boundary is b(^o) = 2. For set of states A = {{zi,M)} 
and state ^o, (Q1a)(?o © t) = | if t<2, and (Q1a)(?o © t) = 1 if t=2. This yields for the 
transition probability from state to A in emb{Z) is: 

Jo 3 3 3 

3. The Product of a CTMC and a DTA 

In this section, we will make the first steps towards the quantitative and qualitative 
verification of CTMCs against linear real-time properties specified by DTA. The aim is 
to computing the probability of the set of paths in CTMC C accepted by a DTA A, i.e.. 
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Pr(C 1= A). We first prove that this question is weh-defined, i.e., that this set of paths 
is measurable. The next step is to define the product of a CTMC C and a DTA A. As 
we will see, this is neither a CTMC nor a DTA, but a mixture of the two. We define the 
semantics of such products and define a probability space on their paths. The central result 
of this section is that Pr(C |= A) equals the reachability probability in the product of C and 
A, cf. Theorem 13.101 In order to facilitate the effective computation of these reachability 
probabilities, we adapt the region construction of timed automata to the product C A, 
and show that this yields a PDP. The analysis of these PDPs will be the subject of the next 
two sections. 

To simplify the notations, we assume w.l.o.g. that a CTMC has a single initial state 
So, i.e., a{so) = 1, and a{s) = for s 7^ sq- The state labels of the CTMC will act as 
input symbols of the DTA. Thus, the alphabet of DTA equals the powerset of the atomic 
propositions, i.e., 2^^. A timed path in a CTMC is accepted by a DTA A if there exists a 
corresponding accepting path in A. 

Definition 3.1 (CTMC paths accepted by a DTA). Let CTMC C = (5, AP, L, sq, P, -E) 
and DTA A = (2^^, Af , Q, qo, Qf, -^)- The CTMC path sq ^ si ^ S2 • • • is accepted by 
A if there exists a corresponding DTA path 

^^_l(£o)Al^ s^c(go,i(so),5o) ^^""'^'^^ > succ[qi,L{si),gi) ••■ 

^ V ' ^ V ' 

=qi =92 

which is accepted by A, where 770 = 0, gi is the (unique) guard in qi such that rji+ti \= gi 
and ?7i+i = {'qi+ti)\Xi := 0], and rji is the clock evaluation when entering gj, for all i. 



3.1. Measurability. The quantitative verification of CTMC C against DTA A amounts 
to compute the probability of the set of paths in C that is accepted by A. Formally, let 

Paths'^ (A) = { pe Paths'^ \ p is accepted by DTA ^ }. 

We first prove its measurability: 

Theorem 3.2. For any CTMC C and DTA A, Paths^{A) is measurable. 

Proof. It suffices to show that Paths'^ {A) can be written as a finite union or intersection 
of measurable sets. The proof is split in two parts: DTA with (1) reachability acceptance, 
and (2) Muller acceptance. The proof of the first case is carried out by (la) considering 
DTA that only contain strict inequalities as guards, (lb) equalities, and (Ic) non-strict 
inequalities. (Note that constraint x = K can be obtained hy x > K f\x > K). 
(la): Let DTA^ A only contain strict inequalities as clock constraints. As all accepting 
paths are finite. Paths'^ (A) = {J^^^ Paths'^{A), where Paths^{A) is the set of paths 
of length n accepted by A. Let p = So si • • • Sn-i > Sn G Paths^{A). Then 

there exists a corresponding path 9 = qo > qi - • • q-n-i > q-a of A which 

is induced by the sequence: 

L(so),go,Xo L{sn~i),gn-i,X„-i ^ ^ 
Qo > 91 • • • Qn-l > Qn, 

with Qn ^ Qf such that there exist {fyi}o^i<n with 1) r/o = 0; 2) rji+ti \= gf, and 3) 
r/j+i = {rji+ti)[Xi := 0], where r]i is the clock valuation when entering qi. 
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We prove the measurabihty of Paths ^{A) by showing that for any path 

p = so s„ G Paths'iiA), 

there exists a cyhnder set C(so, /oi • • •> -^n-i, Sn) {Cp for short) such that: 

p G Cp and Cp C Paths^^{A) for |p| = n. (3.1) 
This is proven in two steps: 

a. (p G Cp.) Let p = sq > G Paths ^iA). We define Cp by considering 

intervals h with rational bounds that are based on ti. Let Ii = \t~ ,tf] such that 
t~ = := tj if tj G Q, and t^ ,tf G Q otherwise, such that: 

ti ^ ^ L^rJ = \4^ = M and t+ - < A. 

where A = min < {77j(x)+tj}, 1 — I {^7(^)+^7} 7^ n with {•} 

denoting the fractional part. Since DTA A only contains strict inequalities, for any i 
with rji+ti 1= Qi, it follows {rii{x)+ti} / 0. 

b. (Cp C Paths^^{A).) Let p' := sq-^ G Cp. Let t?^ := and := 

{r]^+t'j}[Xi := 0]. It remains to show that r/-+t- |= Observe that 7]'q = r]o, and for 
any i > and clock variable x, 

i-i i-i ^ 

j=0 j=0 

Given that guard Qi only contains strict inequalities, it follows rj'-+t'- \= gi. This can 
be seen as follows. Let gi = x > K ioi some natural K. As \rj[{x) — rii{x)\ ^ 
and |i- - ti\ < ^, it follows \{'q'i{x)+t[) - {■qi{x)+ti)\ < A. Note that rii{x)+ti > K, 
and thus r]i{x)+ti — {7]i{x)+ti} = \r]i{x)+ti] > K. Hence, r]i{x)+ti — A > K since, 
by definition, A ^ {Vii^) + ti}. It follows that r]'^{x) + t[ > K. A similar argument 
applies to the case x < K and extends to conjunctions of strict inequalities. Thus, 
■n'i + t'i \= gi, and p' G Paths n{A). 
By (13. ip and the fact that Paths'^{A) Q [Jp^Paths'^ (A) ^P' have: 

Paths'iiA) = IJ Cp and Paths^{A) = \J \J Cp. 

pdPaths^JA) p^Paths^^{A) 

As each interval in Cp has rational bounds, Cp is measurable. It follows that Paths'^ (A) 
is a union of countably many cylinder sets, and hence is measurable, 
(lb): Consider DTA^ A with equalities of the form x = K for natural K. Measurability 
is shown by induction on the number of equalities in A. The base case (only strict 
inequalities) has been shown above. Now suppose there exists an edge e = q "'^'"'^ > q' in 
A where g contains the constraint x = K. Let DTA^ Ae be obtained from A by deleting 
all the outgoing edges from q except e. We then consider the DTA Ae, Af , and Af 
where Ae is obtained from Ae by replacing x = K hy true; A^ is obtained from Ae by 
replacing x = K hy x > K and Af is obtained from Ae by replacing x = K hy x < K. 
Since A is deterministic, it follows that 

Paths^iAe) = Paths^{Ae) \ {Paths'^ {A>) U Paths^{Af)). 
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By the induction hypothesis, the sets Paths^ {Ae), Paths'^ (A^) and Paths'^ (Af) are 
measurable. Hence, Paths^ {Ae) is measurable. Furthermore, as 

Paths'^ {A) = IJ Paths^{Ae), 

e=q "-^'^ > q' 

where all guards g of edge e are equalities, it follows that Paths'^ (A) is measurable. 

(Ic): Let DTA^ A have clock constraints of the form x tx\ K where txiG {>,<}. We 
consider the DTA A= and A^, where A= is obtained from A by changing all constraints 
of the form x txj K hy x = K, and A^ is obtained from A by changing any constraint 
X fx\ K hy x^K, with > = > and < = < otherwise. Clearly, Paths'-' (A) = Paths'^ {A=) U 
Paths'^ (A^). As it was shown before that Paths'^ {A=) and Paths'^ (A^) are measurable, 
it follows that Paths'^ (A) is measurable. 

(2): Let DTA^.A with Qj- = {Fi, . . .,Fk}. Paths^{A) = no<i5Sfc ^a^/^s' where Paths' is 
the set of paths in CTMC C whose corresponding DTA paths are accepted by Fj G Qjr, 
i.e., Paths' = {6* G Paths'^ {A) \ inf(6') = FJ. We have: 

Paths' = fl U U C(so,/o,--- 

71^0 m;S:n SO,...,S„,S„+l...,Sm 

where {s^+i, • • • , Sm} = Lp- with Lj?. the set of CTMC states whose corresponding DTA 
states are Fi, and C{so,Io, • • • , In-i, Sn, ■ ■ ■ , Im-i, Sm) is the cylinder set such that each 
timed path of the cylinder set of the form sq • • • > s„ • • • ^"'"^ > Sm is a prefix 
of an accepting path of A. It follows that Paths' is measurable. Thus, Paths'^ {A) is 
measurable. □ 



3.2. The product of a CTMC and a DTA. A central step in the verification of a CTMC 
C against a DTA A is to construct its synchronous product C (S> A. The resulting object is 
neither a CTMC nor a DTA, but a mixture of the two. We first define this model, called 
deterministic Markovian timed automata, and define a measurable space over its paths. Li 
Section HI we consider the computation of Pr(C |= ^) = Pr (^Paths^ {A)) which is based on 
this product. 

Definition 3.3 (DMTA). A deterministic Markovian timed automaton (DMTA) is a tuple 
M = {Loc,X,iQ,LocF,E,-^), where Loc is a nonempty finite set of locations; X is a 
finite set of clocks; io G Loc is the initial location; Locf is the acceptance condition with 
Locp = Locp ^ Loc the reachability condition and Locp = Locjr C 2^°'^ the Muller 
condition; E : Loc — )• M^o is the exit rate function; and C Loc x CC{X) x 2"^ x Distr{Loc) 
is an edge relation such that: 

(9 9' \ 

I -^A^ ^ and £ -^A^ ^' with g g' \ implies g H g' = 0. 

DMTA closely resemble DTA, but have in addition to DTA an exit rate function that 
determines the random residence time in a location, and an edge relation where the target 
of an edge is a probability distribution over the locations. Concepts such as clock valuation, 

clock constraints and so forth are defined as for DTA. We refer to £ -^^^-e- C for distribution 
C G Distr{Loc) as an edge and to £ ^' > d' with p = C,{1') as a transition of this edge. 



MODEL CHECKING OF CTMCS AGAINST TIMED AUTOMATA 



13 



The intuition is that when entering location the DMTA chooses a residence time which 
is governed by an exponential distribution with rate E(£). Thus, the probability to leave I 
within t time units is 1 — e~^(^)*. Due to the determinism of the edge relation, at most one 

edge, say £ -^'X^ ^ ^ is enabled. The probability to jump to £' via this edge equals C(^')- 
Similar as for DTAs, DMTA"^ and DMTA"^ are defined and DMTA refers to both classes. 

Definition 3.4 (DMTA paths). An (infinite) symbolic path of DMTA M is of the form: 
^0 ^^^^^ h '^^^^^ ^2 • • • where £i (i and pi = Ci(^i+i), for aU i G N. 

A symbolic path induces infinite paths of the form ■ ■ ■ such that 

r/o = 0, {r]i + ti) \= gi, and r/j+i = (r/j +ti)[Xi := 0] where i ^ and rji is the clock valuation 
of <Y in 7W when entering location £{. The path r is accepted by a DMTA^ if there exists 
n ^ 0, such that T[n] € Locp- It is accepted by DMTA'^ if and only if in/(r) G Locjr. 



DMTA semantics. Consider clock valuation r] in location £. As the DMTA is deter- 
ministic, at most one guard is enabled in state {£,r]). The one-jump probability of taking 

the transition £ > £' within time interval / starting at clock valuation 7] in location £, 

denoted prf{£, £',!), is defined as follows: 

p^{£,£',I)= [ £;(£). e-^W^ • Igiv+r) ■ p dr (3.2) 

Jl ^ V ' ^ V ' 

(i) density to leave £ at t (ii)»;+T^g? (iii) probabilistic jump 

Note the resemblance with (j2.ip . Actually, part (i) characterizes the delay r at location £ 
which is exponentially distributed with rate E{£); (ii) is the characteristic function, where 
lg(?7+r) = 1 if and only if rj+r \= g. It compares the current valuation rj+T with guard g 
and rules out those violating g. Part (iii) indicates the probability of the transition under 
consideration. Note that (i) and (iii) are features from CTMCs while (ii) stems from DTA. 
The characteristic function Ig is Riemann integrable as it is bounded and its support is 
an interval; therefore, Prj{£,£',I) is well-defined. The one-jump probability can be uniquely 
defined in this way because it relates to a fixed clock evaluation rj. 

The above characterisation of the one-jump probability provides the basis for defining 
the probability of a set of DMTA paths. Let C{£o, Iq, . . .,In-i,£n) be the cylinder set with 
{£o,---,£n) e -Loc"+^ and Ii C M^g- It denotes a set of paths in DMTA M such that 
for any such path r, r[z] = £{ and r(i) G /«. Let Fv:^ {C{£o, Iq, . . ., In-i,£n)) denote the 
probability of C(£o) -^O) • • -^In-iAn) such that r/o is the initial clock valuation in location £q. 
Let Fi^^ (C(4,/o, . . .,/„_!, 4)) = lPo^(%), where Pf^(r/) is inductively defined as follows: 

1 \i i = n 



Pf^(r/) 



^ ^^(^^)-e-^^^')" - lg,(7? + r)-p, • P^^i^ dT if ^ i < n, (3-3) 




where rj' := {rj + T)[Xi := 0]. Intuitively, P^(r/j) is the probability of the suffix cylinder 
set starting from £i and r]i to £n- It is recursively defined by the product of the probability 
of taking a transition from £i to ^j+i within time interval Ii (cf. (*) and (j3.2p ) and the 
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probability of the suffix cylinder set from and r]i+i on (cf. For the same reason 

as pri{i,i',I) is well-defined, P^(r/) is well-defined. 

Example 3.5. The DMTA^ in Figure 5(a) has initial location Iq with two outgoing edges, 
with guards x < 1 and 1 < x < 2. We use the small black dots to indicate distributions. 
Assume t time units elapse in £o. If the current clock evaluation rj satisfies rj(x) < 1, then 
the upper edge is enabled and the probability to go to ii within time t is pgiiojii, [0, t]) = 
(1 — e"'"''*)-!, where E{£q) = ro; no clock is reset. It is similar when 1 < ri{x) < 2, except 
that X will be reset (cf. the lower edge emanating from location £q). If r/(x) ^ 2, no outgoing 
edge is enabled, and the DMTA stays in Iq ad infinitum. 




l<x<2,{x} 



1<2;<2,{3;} 



(a) DMTAO C(g)A 



r2 {b} 

''o 0.5 n o^3~~X''*-'^ 1~} 

ra {c} 



(b) CTMC C 



{a}, X < I, 




ib},x > 1,0 



{a},l <x < 2,{.x} 



(c) DTA'* A 



Co,0<;I<l 


5 


Co,ls;.r<2 








' s 


. reset, 0.5 






^i,1^3:<2 





f4,0 



reset, 0.2 




•"5, ''2 



C2,1<.T<2 


s 


t2,X > 2 






1 

l'7,0 


1 


4,l^a:<2 


s 





(d) Reachable region graph oi C ® A 



Figure 5: Example product DMTAO of CTMC C and DTAO A 



3.3. Product DMTA. The product C ^ for CTMC C and DTA A, is a DMTA. 

Definition 3.6 (Product of CTMC and DTA). Let C = (S, AP, L, sq, P, -E) be a CTMC 
and A = (2^P,;f,Q,go,QF,^) be a DTA. Let C ^ = (Loc, A', 4, Locp, -E, -->) be the 
product DMTA, where Loc = S x Q; £q = {so,qo); E{{s,q)) = E{s); and 
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• LocF = Locp '■= S X Qf, if Qy = Qf (reachability condition) 

• Locf = Locjr := U^gQ^ S X F, if Qf = Qj^ (Muller condition) 

and ~^ is defined as the smallest relation defined by the rule: 
P(s,s') >0 A q ^W'3'^> r^ 



{s,q) — s^C 



such that C((s',g')) = P('S,s')- 



The DMTA C ® A is basically the synchronous product of CTMC C and DTA A such 
that transition s — > s' in C is matched with the edge q ^^^^'^'^ > q'^ i.e., the set of atomic 
propositions of s acts as input symbol for the edge from location q to q' in A. The probability 
of the joint evolvement of C and A is given by P(s, s'), the discrete probability of s — )• s' in 
C, whereas the residence time in the location {s,q) is given by E{s), the exit rate of s in C. 
It is easy to see from the construction that C iSi A is indeed a DMTA. The determinism of 
the DTA A guarantees that the induced product is also deterministic. In C (8) from each 
location there is at most one "input symbol" possible, viz. L{s). For the sake of convenience, 
input symbols can be omitted from C (Si A. 

Example 3.7. Let CTMC C and DTA^ A be given in Figure [5(b) and 5(c) , respectively. 
The product DMTA"^ C(SA is depicted in Figure [5(a)| Since Qp = {qi} in A, the set of 
accepting locations in DMTA^ is Locp = {(■52,91)} = {^s}- 

Example 3.8. For the CTMC C in Figure |6(a)| and the DTA"^ A in Figure |6(b)| with 
acceptance family Qjr = {{qi,q2},{Q3,Q'i}}, the product DMTA"^ C (g) ^ is shown in 



Figure |6(c)[ Locjr = {{(si, gi), (sj, 92)}, {(s^, gs), (sj, 94)}}, for any Si, s'i, sj, s'j G S, i.e., 

L0CT = {{ll,i2,i3}, {^4,4,4}}. 

The set of accepted paths in DMTA is defined by: 

AccPaths^^^ := {t e Paths^^-^ \ T is accepted by Ci^A }. 

For n-ary tuple J, let J[i denote the i-th entry in J, for 1 ^ i ^ n. For a (C^A)- 
path r = (so, qo) (si, qi) ■ ■ ■ , let tIi := sq si • • • , and for any set 11 of 
(C(8)^)-paths, let n|,i = Uren''"!-!- The following lemma asserts that there is a one-to-one 
relationship between paths in CTMC C accepted by DTA A and accepting paths in C iS) A. 

Lemma 3.9. For any CTMC C and DTA A, Paths^{A) = AccPaths^'^-^ii. 

Proof. We provide the proof for DTA^ A; the proof for DTA"^ A is similar. 

(C) Let p G Paths^{A). We prove that there exists a path r G AccPaths^®-^ with p = t[i. 
Assume w.l.o.g. that p = sq -^si • • • s„_i > 5^ € Paths'^ (A), i.e., s„ G Qf, Vo \= 0, 
and for ^ i < n, rji+ti \= gi and 77^+1 = {rii+ti)[Xi := 0], where rji is the clock valuation 
in A when entering state Sj in C. We construct a timed path 9 G Paths'^ from p such that 

9 = qo > qi - ■ ■ qn-i > qn, where the clock valuation on entering Si and qi 

coincides. From p and 9, we can now construct the path 

T = (so, go) {Sl,qi) ■ ■ ■ (s„-l,g„-l) ^"'^ > {Sn,qn), 

where {sn, qn 

) G Locp. It follows that r G AccPaths^'^'^ and p = rti- 
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(c) DMTA" C(g)A" 
Figure 6: Example product DMTA^ of CTMC C and DTA"^ A'^ 

Q) Let T e AccPaths^'^-^. We prove that r|,i G Paths'^ {A). Assume w.l.o.g. that 
r = (so, go) ^ • • • {sn, Qn) G AccPaths^^-^ , 

with {sn,qn) G LocF, r?o N 0, and for ^ i < n, 7]i+ti \= gi and r/j+i = (7/j+tj)[Xj := 0], 
where r]i is the clock valuation when entering location {si,qi). It then directly follows that 
Qn ^ Qf and t|,i G Paths'^ (A), given the entering clock valuation r]i of state Sj. □ 

Theorem 3.10. For any CTMC C and DTA A, 

Pt^ (Paths^iA)) = Pi'^^^-^ {AccPaths^^-^) . 

Proof. We provide the proof for DTA^ A; the proof for DTA'^ A goes along similar lines 
as in the proof of Theorem I3.2i 

According to Theorem 13.21 Paths'^ (A) can be rewritten as the combination of cylinder 
sets of the form C{so,Io, ■ ■ ■ ,In-i,Sn) which are all accepted by DTA^ A. Note that this 
means that each path in the cylinder set is accepted by A. By Lemma 13.91 namely by path 
lifting, we can establish exactly the same combination of cylinder sets C{£o, Iq, . . . , /„_!,£„,) 
for AccPaths^®'^ , where Sj = It then suffices to show that for each cylinder set 

C(so, Iq, • • • , /n-i) Sn) which is accepted by A, Pt^ and Px^®-^ yield the same probabilities. 

For the measure Pr^, according to Eq. (j2.ip (cf. page[5|), 

Pr^(C7(so,/o,...,/n-i,s„)) = J] j V{si,Si+i)-E{si)-e-''^'^^^dT. 
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The measure Pr^®'^, according to SectionE^l is given by ¥^^-^{0), where P^;®-^(??) = 1 
for any clock valuation 77 and for any ^ i < n: 



where rji+i = {rn + Ti)[Xi := 0] and Ig^irji + n) = 1, if r]i + n \= gf, 0, otherwise. 

We will show, by induction, that Pf®"^(r?i) is a constant, i.e., is independent of rji, if the 
cylinder set C{£o, Iq, . . . , In-i,(-n) is accepted hy C®A. First note that for this cylinder set 
there must exist some sequence of transitions 

„ 90,^0 „ „ g„_i,X„_i 

with ?7o = and Vtj G Ij with ^ z < n, r/j + ^ (7i and T^j+i := (r/j + ti)[Xi := 0]. 
Moreover, according to Definition 13.61 we have: 

Pi = P{si,Si+i) and ^(^i)=^(si)- (3.4) 
We apply a backward induction on n down to 0. The base case is trivial since P^®-^(r/„) = 1. 
By the induction hypothesis, P^_^{^(r/j+i) is a constant. For the induction step, consider 
i < n. For any Ti G li, since rji + Ti \= gi, 'i-gi{i]i + Tj) = 1, it follows that 



I.H. 



Eq.llTill 



[ F{s„s,+,)-E{s,ye-''^'^^-^dn-¥^^,^{r,,+i). 



Clearly, this is a constant. It is thus easy to see that 



which completes the proof. □ 



3.4. Region graph construction. Theorem 13.101 asserts that the probability of CTMC 
C satisfying the DTA specification A equals the reachability probability of some accepting 
location in C <Si A. The state space of C (8> however, is infinite. As a next step towards 
obtaining an effective procedure for computing reachability probabilities in C <^Awe adopt 
the standard region construction of timed automata [T] to DMTA. This yields a stochastic 
process, namely a PDF. Here, we consider the region construction for finite acceptance 
conditions, i.e, DMTA^. The details for DMTA'^ are slightly different (only the acceptance 
set differs) and are provided in Section [H 

Let us briefly recall the concept of a region. Formally, a region is an equivalence under 
=, an equivalence relation on clock valuations. A region is characterized by a specific form 
of a clock constraint. Let Cx^ be the largest constant with which Xi (z X is compared in 
some guard in the (DM)TA. Clock evaluations r/, r/' G V(^) are clock- equivalent, denoted 
r] = r]', if and only if either 

(1) for any x G ^ it holds that 7]{x) > Cx and r]'{x) > Cx, or 



18 



T. CHEN, T. HAN, J.-P. KATOEN, AND A. MEREACRE 



(2) for any Xi,Xj G X with r]{xi),r]' {xi) ^ and r]{xj),r]'{xj) ^ Cxj it holds: 
[r]{xi)\ = [7]'{xi)\ and {r]{xi)} ^ W{xi)} iff vi^j) ^ r]'{xj), 
where [dj and {d} are the integral and fractional part of d G M, respectively. 

This clock equivalence is coarser than the traditional definition by merging the "bound- 
ary" regions (those with point constraints like "x = 0") into the "non-boundary" regions 
(those only with interval constraints like "0 < y < 1"). For instance, for X = {xi,X2}, the 
boundary regions {xi = 0,X2 = 0), (0 < xi < 1, X2 = 0) and (xi = 0, < X2 < 1) are merged 
with the non-boundary region (0 < xi < 1, < X2 < 1) yielding (0 ^ xi < 1, ^ X2 < 1). 
The reason for this slight change will become clear later. 

Let TZe{X) be the set of regions over the set X of clocks. For G,0' G TZe{X), B' is 
the successor region of if for all ?? ^ there exists 6 G M>o such that r]+5 \= Q' and 
\/5' < 5. r]+6' 1= G V ©'. The region Q satisfies the guard g, denoted @ \= g, i& Vrj \= Q. 
r] \= g. The reset operation on region is defined as Q[X := 0] := {r][X := 0] \ rj \= ©}. 

Definition 3.11 (Region graph of DMTA^). The region graph of DMTA*^ A4 = {Lac, X , £q, 
Locf,E,'^) is G{M) = (y, Wo, Vf,A, "-^►), where 

• V = Lac X lZe{X) is a finite set of vertices with initial vertex vq = (^O)O); 

• Vp = {v \v[i ^ Locp} is the set of accepting vertices; 

• A : V ^ M^o is the exit rate function where: 



A{v) 



E{v[i) if V ^ v' for some v' €V 
otherwise. 



• ^ C y X (([0, 1] X 2'^) U {6}) X y is the transition (edge) relation, such that: 

► V v' if v[i = v' [i, and v' [2 is the successor region of v I.2; 

► V ^ if v[i I ^'^> v'li with v[2 \= g, and vi2[X := 0] = v'[2- 

Any vertex in the region graph is a pair consisting of a location and a region. Edges of 

S , p,X 
the form v ^ v are called delay edges, whereas those of the form v ^ v are called 

Markovian edges. Note that Markovian edges emanating from a boundary region do not 

contribute to the reachability probability as the time to hit the boundary is always zero 

(i.e., \){v,rj) = in Eq. (14. 3p . page l20p . Therefore, we can safely remove all the Markovian 

edges emanating from boundary regions and combine each such boundary region with its 

unique non-boundary (direct) successor. In the sequel, by slight abuse of notation, we refer 

to this simplified region graph as Q{Ai). Note that then u|,2[-'^ := 0] C v'l2 in the last item 

of Definition 13.111 

Remark 3.12. [Exit rates] The exit rate A(v) equals if only delay transitions emanate 
from V. The probability to take the delay edge within time t is e"^^"^* = 1, while the 
probability to take Markovian edges is 0. 



Example 3.13. For the DMTA^ C^A in Figure 5(a) , the reachable part (forward reachable 



from the initial vertex and backward reachable from the accepting vertices) of the simplified 



region graph Q{C^A) is shown in Figure 5(d) Note that the exit rates on and vj are 0, 



as only a delay edge is enabled in these vertices. 
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The following result asserts that the region graph obtained from a DMTA is in fact a 
PDP. This is an important observation, as verification now reduces to analyzing this PDP. 

Lemma 3.14. The region graph of any DMTA induces a PDP. 

Proof. Let DMTA^ M = {Lac, X, 4, Locf,E, with region graph g{M) = {V, vq, Vf, A, 
Define Z{A4) = {V, X, Inv, (f), A, /x) where for any v (zV: 

• Inv{v) := v[2 and the state space S := \ v ^V,ri \= Inv{v)^\ 

• 4'{v,r],t) :=r] + t; 

• A{v,r]) := A{v); 

• if u ^ in g{A4), then fi{{v , r]) , {{v' , rj)}) := 1, provided r] \= dlnv{v)] 

• if ti ^ f ' in g{A4), then fi{{v , rj) , {{v' , r][X := 0])}) := p, provided rj \= Inv{v). 

It fohows directly that Z{M) is PDP. □ 

Note that the acceptance conditions play no role in the definition of a PDP, thus this lemma 
applies to both DMTA<^ and DMTA'^. 



4. Verifying CTMCs Against Finite DTA Specifications 

The characterization of the region graph oi C A as a PDP paves the way to the 
verification of CTMC C against DTA^ specification A. This section concentrates on the 
quantitative verification problem and deals with single-clock DTA separately. 



4.1. Quantitative verification with arbitrarily many clocks. The central issue in 
quantitative verification is to compute the probability of the set of paths in C accepted 
by A. By Theorem 13.10^ this is equal to computing reachability probabilities in DTMA 
C ® A. The remaining question is how to determine these probabilities. To that end, we 
show that this amounts to determine reachability probabilities of untimed events in the 
embedded PDP Z{C ® A) (cf. Theorem 14.31 below) . These probabilities are characterized 
by a Volterra integral equation system of second type. As solving this integral equation 
system is typically hard, we present an effective approximation algorithm. 



Characterizing reachability probabilities. We first consider determining unbounded 
reachability probabilities in the PDP Z = Z{C ® A). This is done by considering its 
embedded PDP, the DTMP emb{Z), as for unbounded reachability probabilities, the timing 
aspects are not important. Note that the set of locations of PDP Z and emb{Z) are equal. 
Besides, the discrete probabilistic evolution of Z and emb{Z) coincide. The main difference 
is that enib{Z) is time-abstract whereas Z is not. 

Let initial state (fo,0) and T C y be the set of goal locations. For state {v,rj), let 
Pro6'^™^^^^ ((iJ, r/), r) , Proh^{r],T) for short, denote the probability to reach some state in 
(T, •) from state [v, rj) in emb{Z). These probabilities are recursively defined as follows. For 
vertex u G we have: 

[1 if e T 

Proby{r],T) i Prob^s{'n,T) + J2p,x Prob^^i{r],T) otherwise ^^'^^ 

The case v £ T is evident. In case v ^ T, then either a delay can take place (first summand), 
or a Markovian edge is taken to vertex v' (second summand). 
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For a delay transition v ^ v' we have: 

Proh,4ri,T) = e-^(")-^(''''?) •Pro6,,(?7+b(f,??),r), (4.2) 

where e^^'-^^'^^"'''^ is the probabihty to stay in v for at most b(t',??) time units. Recall 
that \>{v,r]) is the minimal time for state {v,r]) to hit the boundary dlnv{v). Stated in 
other words, e"^^*'^'''^''''') is the probability to reside in v without violating the invariant. 
The reachability probability from the resulting state r/+b(u, rj) is then given by the second 
multiplicand in Eq. (|4.2p . This equation is based on Eq. (|2.3p by determining the multi-step 
reachability probability using a sequence of one-step transition probabilities. 

f , we have: 

p-A{v)-e'^^^>^ ■ Prob,,{{ri + T)[X := 0],r) dr. (4.3) 



Here, A{v)-e~^^^'^"^ denotes the density to stay for exactly r time units in v. As any delay 
up to b(u, ry) does not violate the invariant, r ranges over the dense interval [0, b(v, rj)]. The 

state after first delaying r time units and then taking the edge v ^ v' is {rj + t)[X := 0]. 
Eq. (fO|) is derived from Eq. ([221). 



For the Markovian transition v 

Prob^^^,{r],T) 



X2 > 1, {xi} 



ri 



4=(so,'7o) 



4 = (gi,gi)J) 





1 




<2,{a;2} 






(a) 


DMTA'* C ® 


A 




i>o,0 












5 


io 


6 




0^a;i=.T2<l 











V3,0 



0^a;i<l 
l^a;2<2 

X2>Xi + l 



Vi,0 



0^xi<l 

X2>Xi + 2 



(b) Reachable region graph Q{C ^ A) 
Figure 7: Reachable fragment of its region graph 



Example 4.1. Consider the DMTA^ in Figure 7(a) and its region graph in Figure 7(b) Let 
T = Vphe the set of goal locations, i.e., the set of target states {{v, rj) \ v £ Vp, rj \= Inv{v)}. 
The system of integral equations for vi in location Iq is as follows. For 1 ^ xi = 2:2 < 2: 

Probv^{xi,X2) = Proby^^s{xi,X2) + Probv^^v-A^ii ^2), 

where 

Prob,,4xi,X2) = e-('-"i)'-"-Pro6,,(2,2) 
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and 

f-2-xi 

Prohv^^y.j^{xi,X2) = I ro-e~''"^-Pro6t,3(0,X2 + r) dr 
where Prob^.^{0, X2 + r) = 1. The integral equations for vertices t'2,f4 are similar. 

Remark 4.2. Clock valuations rj and rj' in region G may induce different reachability 
probabilities. This is due to the fact that r] and rj' may have different periods of time to 
hit the boundary, Thus, the probability for rj and rj' to either delay or take a Markovian 
transition may differ. This is in contrast with timed automata, as well as probabilistic 
extensions thereof [22], where clock valuations in the same region are not distinguished. 

Hence, reachability probabilities in the embedded PDP of Z{C(S)A) are characterized by 
a system of Volterra integral equations (j4.ip . One can read (|4.ip either in the form /(^) = 
/z)om(5) '^')-^(^'^')' where K is the kernel and Dom{£^) is the domain of integration 
depending on the continuous state space S; or in the operator form /(^ = [J f ){£,)■, where 
J is the integration operator. Generally, ()4.ip does not necessarily have a unique solution. 
It turns out that the reachability probability Proby^iO) coincides with the least fixpoint of 
the operator J' (denoted by lfpi7) i.e., Pro6^y(0) = (Ifp J7) (f o , 0) . 

Theorem 4.3. For any CTMC C and BTA^ A, 

Fr9^-^{AccPaths^'^-^) is the least solution of Prob'^^{6,VF), 

where DTMP V = emh{Z{C ® A)). 

Proof. Let Vr^'^ (^AccPaths^®"^^ be the least solution of the system of integral equations: 
1 if £ G Locp 

Pr(£,r?)=<( / E{e)-e-^^^^^ ■ Ig (ry+r) -p- Pr(f, (r/+r)[X := 0]) dr otherwise, 

e 

Informally, Pi{£, rj) is the probability to reach the set of locations Locp from location d. and 
clock valuation r/. The above integral can be simplifed as follows. W.l.o.g. assume clock 
constraints to be of the form x < c, where c G N and <G {<,<,>,>}. Then we have: 

Pr(^,7y) = / ' E(£)-e-^(^)^ • ^ p- Pr(^', (r/+T)[X := 0]) dr, 



p 

where ti,t2 G Q^o U {oo} and rj+r \= g for any ti < t < t2- 

If £ G Locp, the theorem follows directly. In the remainder of the proof, assume 
£ ^ Locp- Our proof is based on showing that for any I ^ Locp and clock valuation r/, 

Pr(£,r/) = Pro6,„(r/,V>), (4.4) 

where vq is the initial vertex in the region graph Z{C ® A) with foil = and = {u G 
y [ wti G Locp}- This is done as follows. For natural n, let Pr"(^, r/) be the probability 
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to reach Loop in n steps in C ^ A. For n = 0, we have Pr"(^, r/) = 1 ii £ G Locp and 0, 
otherwise. For n > 0, we define inductively: 

Pr'^(£,r?) = r'^(^)-e-^(^)" • ^ p ■ Fr^'-^l' ,7]') dr. 

J tl 

Similarly, let Proh'^{ri,VF) be the probability to reach the set of goal states in n > 
steps: 

^ ^nr j Problsiv,Vp) + Probr{7^,Vp), ifv^Vp 

Prob'i{r],VF) = < ' (4.5) 

I 1, otherwise 

Probl'^{v,VF) = t^Tivye-^'-''^^- p-Proby\{r]+T)[X:=0],VF) dT, (4.6) 
Jo ^ 



p,X 



Probls{v,VF) = e-^^^^^^'") •Pro6^,(r/ + b(7;,r/),yi.). (4.7) 
In the sequel, we show that for any n G N, it holds: 

Pr^{i,r^) = Probl{r^,VF). (4.8) 

The theorem then follows from the fact that lim Pr"'(£,r]) = Pr{i,r]) and, similarly. 



lim Prob'i{r],VF) = Prob^{r],VF). 



.«o=(^,eo) 



Vm-l = {i,&rn-l) 
\>{v,n^l,fim-l) = l 



s 




s 


s 






K'"m,'?m) = l 







p,X 



p,X 



Figure 8: The sub-region graph Z[C ® A) for the transition from £ to £' . 

The proof of Pr"'(^, rj) = Prob^^{r], Vp) is by induction on n. 

(1) (Base case.) For n = 0, Pr^(^, r/) = = Prob^^['q, Vp) ii i ^ Locp, and 1 otherwise. 

(2) (Induction step.) Consider n+1. Let edge £ ( in C A. Assume the fragment of 
the region graph Z{C (S) A) that corresponds to this edge with ({i, £') > is as shown in 
Fig. [SI Location £ induces the vertices {vi = (^, Gj) | ^ i ^ k). Intuitively speaking, 
the transition from location £ to £' is enabled in region 0j for m ^ i ^ k, whereas only 
a delay can take place in all regions 0j with i < m (while staying in location £). 

Let rji be the clock valuation when entering vertex Vi, i.e., fjQ = -q and fji = + 
b(uj_i, r}j_i) for < i ^ fc. It is assumed that f/j ^ where g is the guard of the edge 
at hand, for i < m and i > k. Accordingly, 

m—l k 

tl = ^ b(t;i,57i) and t2 = y^^\){vi,f]i) 

i=0 i=0 

are the lower and upper bound, respectively, of the interval during which guard g holds. 
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For convenience, let p"(r/) := Prob^ g{r],VF) + Probl'"'{r],Vp). Given the fact that 
only a delay transition can be taken before time ti, it holds that 

Ko^H^) = e-'^^^-'^^-p:^\fim), where 

p:::\Vm) = Prob:^^{flm,VF)+ProbZ^\flm,VF). 

We now derive: 



-tiA(uo). 



/ A(i;^)-e"^('^™)^- p-Prob^,J{fim+T)[X ■.= 0],VF)dT 

rti+b(Vm,flm) 

/ A(i;^)-e-^(^-)^- V p-Prob^, {{fi^+T-h)[X ■.= 0],VF)dT. 



p,X 



Now consider: 

Using the definition of Prob2'^^g{fim, Vp) (see Eq. (|4.7p ). together with the result derived 
above, yields the following sum of integrals: 

i=0 •^*l+E}=oK«m+j,'?m+i) 

i-1 

Ep-Prob'^, ({fjm+t+r-ti-y^\>{vm+j,'nni+j))[X := 0],Vf) dr. 



=F"{t) 

Using F"-{t) we obtain: 



Pv^Hv) = r A(^^o)-e-^(^°)^-F"(r)dr. (4.9) 



Notice that 

m—l i—1 
j=0 j=0 

^ V ' 

= tl 

Therefore, for any t e[ti + Yl'j=o K'^m+j,'nm+j),h + YTj=o K^m+j, 5?m+i)], if^k-mwe 
obtain 

i-l 
j=0 

From the induction hypothesis (for n), it follows that Pr"(^, r?) = Prob^g{ri,VF) with 
vq [i = I. Therefore, for any t G [ti + X;}=o K^^m+j' 'Vm+j)' *i + I]j-=o K^m+j, ??m+j)] and 
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I'm+i [i = i k — m, we get 



i-1 



^ p-Pr"(/,(,?+t))[X:=0]) 



^ p.Pr"(^',(r? + t))[X:=0]). 

Substituting this result into equation (|4.9p results in 

P'i^Hv) = r A(^).e-^W-- p.Pr"(/, (r?+T))[X := 0])dr. 

As for t;o ^ Vp, Prob'^+\r],VF) = P^^Hv) we get that Pro6^+^(r?, Vp) = Pr"+i(^,»?)- 

□ 

Note that, similar to the computation of reachability probabilities in DTMCs |18j . the 
goal states in T C S as well as all states that cannot reach T can be made absorbing, i.e., 
all outgoing edges can be removed, without affecting the reachability probabilities. This 
may yield a substantial state-space reduction. 

Approximating reachability probabilities. The results so far assert that Pr(C j= A) 
coincides with reachability probabilities in an embedded PDP that is obtained via a region 
construction applied on the product C(i^A. The previous result shows that such reachability 
probabilities are characterized by Volterra equations of the second type [2]. Such integral 
equation systems can be solved using techniques explained in standard textbooks, such 
as [12]. An alternative option — inspired by a formulation of bounded reachability prob- 
abilities in arbitrary PDPs [16] — is to approximate the probability Pr (^Paths'^ (A)) by a 
system of partial differential equations (PDEs, for short). The intuition is to consider paths 
that are accepted within some time bound tf. Let DTA A[tf] be obtained by adding a single 
fresh clock z, say, to DTA A which is never reset, and strengthening all guards of incoming 
edges into g G by adding the conjunct z ^ tj. Obviously, Paths'^ {A[tf]) C Paths^ (A). 
Note that lim Pr(Pat/i/(^[t/])) = FiiPaths^ (A)). 

Given CTMC C, DTA^ A, time bound tj and PDP Z{C (E) A) = {V, X , Inv, cf), A, fi), 
we have: 

FT^{Paths^{A[tj])) = J2 f KitfAdii), 



vGVf 
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where h^^{tf,0,r]) is the probabihty to reach the state {v,r]), with v £ Vf and r] \= Inv{v) 
at time tj from state (^0,0). The transition probabihty function hH^^ltjjO,?]) is described 
by the following equations: 

• for v € y \ Vf, V G Vf with r] \= Inv{v), f] \= Inv{vf) and y € (0,t/): 

^%^ + E^%|^ + A(,).2: p.(r«te..,[X:=01,,)-njfe,,,,7)) = 0. (4.10) 

1 = 1 ^ P,-^ , 

where r/(*) is the i'th clock variable. 

• K!^{0,rj,f]) = 1, when f = ?; and rj = fj, h^{0,r],f]) = 0, otherwise. 

• the boundary conditions are: for v,v V, 7] \= dlnv{v),fi \= dlnv{v) and transition 

V ^ v' we have h'^{y, r], fj) = ??, ?7)- 

Equation ()4.10p is obtained by simplifying a corresponding characterisation in Davis |16j . 
where the author defines the function ^^(•) as an expectation. In our setting, h^y^^{tf,{),ff) = 
E[l(Xj^)|Xo = where Xj- is the underlying stochastic process of the PDP Z with the 
state space S, ^ = (f,0) and l{Xtj-) is the characteristic function such that l{Xtj) = 1 
if and only if Xt^ = {v,f]). The PDE (j4.10p is a special case of [16j as the flow function 
in Z is linear and the probabilistic jumps to the continuous part of the state space S are 
non-uniform. 

4.2. Single-clock DTA*^ specifications. For single-clock DTA*^ specifications, we can 
simplify the system of Volterra integral equations (of second type) obtained in the previous 
section. As we will show in this subsection, the probability that a CTMC satisfies a single- 
clock DTA is given by a system of linear equations whose coefficients are a solution of a 
system of ODEs that can be solved efficiently. The key observation is that the region graph 
corresponding io C(^A can be naturally divided into a number of subgraphs, each of which 
is a CTMC. 

Let ^ be a single-clock DTA with finite acceptance criterion, and {cq, . . . ,Cm} be the 
set of natural numbers that appear in the clock constraints of A. Assume = cq < ci < 

• • • < Cm, and let Acj = Cj+i — c, for ^ i < m. Note that for single-clock DTA, 
the regions in the region graph oi C ® A can be partitioned by the following intervals: 
[co, ci), [ci, C2), . . . , [cm, 00). Using this observation, we partition the region graph Z{C A) 
as follows. 

Definition 4.4 (Partitioning of region graph). Let Q{C ® A) = (y, i;o, Vf, A, "^), or Q 
for short, for single-clock DTA^ A. The partitioning of Q is defined as the collection of 
subgraphs Qi = (V^, V^,, Aj, ^j), for ^ i ^ m where: 

. v^ = {(^,G) GV [ec [c„Q+i)} 

• Vf, = Vi^^ Vf, 

• Ki{v) = K{v) if u G Vi, and otherwise, and 

• ^ = [J Mj U U -Bj, where 

— Mi is the set of Markovian edges (without reset) between vertices in V^, 

— Fi is the set of delay edges between Vi and V^+i, 

— Bi is the set of Markovian edges (with reset) from to Vq. 
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l<;r<2 


/ 5 \ 


t-i,x > 2 
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\ 



\ 



6o Q\ 02 

Figure 9: Partitioning the region graph of Figure [5(d) | 

Since the initial vertex of Qq is vq and the initial vertices of Gi for < i ^ m are 
implicitly given by the edges in -Fi-i, we omit them. Note that the subgraph Qm involves 
only infinite regions and has no outgoing delay transitions. 



Example 4.5. Consider the region graph in Figure 5(a) (page [2]). The partitioning of 
this region graph is depicted in Figure [H The edges in Mj , Fi and Bi are labeled with 
probabilities, 6 (delay), and "reset" with probabilities, respectively. Observe that if v = 
(£, [cj, Cj+i)) S Vf, then v' = (£, [cj, Cj+i)) G Vp for i < j ^ m. (In this example, this 
applies to V = V7 and v' = wg-) This is true since Vp = {(^, true) | £ £ Locp}. Thus, from 
any final vertex in Vi with i < m, there is a delay transition to the next region (if any). 

Assume \Vi\ = ki. We now define for each type of edge {M,B, or F) a matrix (M, B, 
and F, respectively). Let x G M with x G [0, Acj]. Then: 

• Dj(x) G is the delay probability matrix, where for any ^ j ^ /cj, Dj(x)[j,j] = 

^-E{vl)-x g^j^^ off-diagonal elements are zero. 



Mi{x) = Di(x)-E,-P, G 



is the probability density matrix for Mj-edges, where 



and Ej are the transition probability matrix and exit rate matrix respectively, for vertices 
in V^. 

• Bj(x) G M'=i^*^o ig ii^Q probability density matrix for the -Bj-edges, where Bi{x)[j,k] indi- 
cates the probability density function to take a Sj-edge from G Vi to v'^ G Vq. 

• F, G ^^i^^i+i is the incidence matrix for F,-edges, i.e., Fi[j, k] = 1 if and only if there is 
a delay transition between G Vi and v'' G V^+i. 

Due to the fact that in any subgraph Qi there are only Markovian jumps without resets, 
and no delay transitions, the subgraph {Vi, Ai, Mi), i.e., Q restricted to Markovian jumps 
(without resets) forms a CTMC Cj, say. To take the effect of Markovian jumps with resets 
into account, we define for each Gi the augmented CTMC C? with state space V^U Vq, where 
all Vb-vertices are absorbing, i.e., do not have any outgoing edges. The edges connecting Vi 
to Vq are kept. The augmented CTMC is used to calculate the probability to start from a 
vertex in Gi and take a reset edge within a certain period of time. 
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Figure 10: CTMCs corresponding to the (augmented) subgraphs 

Example 4.6. Consider the partitioned region graph in Figure [9l The matrices for Qq are: 

\ / 1 

Mo(x) = I 0.5Tre-''i^ 0.2Ti-e-'"i^ Fq = 1 
/ VOOlO 



The matrices for Qi and its augmented version are given by: 
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The corresponding CTMCs and their augmented version are depicted in Figure [101 
For CTMC C with k states and rate matrix E • P, let: 

n{x)= [ M(r)n(x -r)dr + D(x). (4.11) 

Intuitively, n(x)[j, m] indicates the probability to move from vertex j to m at time x. 
The following proposition states the close relationship between 11 (x) and the transient 
probability vector of C. Let p{t) be the transient probability vector where Ps{t) is the 
probability to be in state s at time t given the initial distribution a. 
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Proposition 4.7. Given a CTMC C with initial distribution a, rate matrix E • P and 
n(t), p(t) satisfies the following two equations: 

Pit) = a-n(*), (4.12) 

= P(t)-Q, (4.13) 

where Q = E-P — E is the infinitesimal generator. 

Equation (|4.13p is the well-known forward Chapman-Kolmogorov equation. According 
to this proposition, solving the integral equation for 11 (t) boils down to solving the system 
of ODEs (|4.13|) given some initial distribution vector a. This can be done using standard 
means for CTMCs such as uniformization. ^ 

Now let the probability vector Ui{x) = [uj{x), . . . ,u^'{x)] £ R.^^^^ where n^(x) is the 
probability to move from vertex vj € Vi to some vertex in Vp (in G) at time x. Based on 
the equations (|4.ip - ()4.3p . we provide a set of integral equations for Ui{x) which later on is 
reduced to a system of linear equations. Distinguish two cases: 

Case ^ i < m: for 

Ui{x) = / ' mi{T)U^{x+T)dT+ / ' Bi(T)dT-C7o(0)+D,(Ac,-x)-FiC7,+l(0), (4.14) 
JO Jo 

where x G [0, Acj]. Let us explain this equation. The last summand is obtained from (14. 2p 

where Dj(Acj— x) is the probability to delay until the "end" of region i, and FjC/j_|_i(0) 

denotes the probability to continue in Gi+i (at relative time 0). Similarly, the first and 

second summands are obtained from (j4.3p : the former reflects the case where clock x is not 

reset, while the latter considers the reset of x (thus, implying a return to ^o)- 

Case i = m: 

poo poo 
tj^ix) = / m^{T)Um{x+T)dT + If+ B„(T)(iT • C7o(0) (4.15) 

Jo Jo 

where for x G [cm,oo), Mm(T)[u,-] = Mm(T)[^;,-] for v ^ Vp-, otherwise, and Ip is the 
characteristic vector for Vp- Note that Ip stems from the second clause of ()4.ip . and M. 
is obtained by setting the corresponding elements of to 

Example 4.8. The matrices for Q2 are given as: 

r2-e-^2^ \ £, 1 



m 



Ui{x) 


n»(x) 








M2(x) - > Q Q / - - V 
For augmented CTMC , let 

n?(x) 

where S R'^o^'^* is the zero matrix and I € M'^o^'^o is the identity matrix. Matrix 11 j 
indicates the transient probabilities for the CTMC Cj. Intuitively speaking, II? contains 
the probabilities starting from Vi and ending in Vq. 

Theorem 4.9. For subgraph Qi (with ki vertices) of Q, it holds that: 

.^^^ _ r ni(Aci)-F,-J7,+i(0)+n^(Aci)-?7o(0) if i < m 

\ Pm ■ Um{0) + If + Bra ' Uo{0) if i = m 
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where Pi{v,v') = Pi{v,v') if v ^ Vp; otherwise and Bm = /q°° Brrt(T) dr. 
Proof. Distinguish two cases: i < m and i = m. 

(1) (i < m.) Consider the augmented CTMC Cf with kf = ki + /cq states. From equa- 
tion (I4.14p . and the fact that C? contains reset edges of Cj, we have: 

Unx)= T'" 'M^(r).?7f(x+r)dr + D^(AQ-x)•F^^7,(0) 
Jo 

f U (x) \ 

where Uf{x) = — -^^ ) € M'^^'^^, U'Ax) € M'^"^-'^ is the vector representing the reach- 
abihty probabihties for the augmented states in Qi, Ff = ( | B- ) € Kfc»''x(fci+i+'=o) 
such that F^ = ^ ^ € is the incidence matrix for delay edges and B- = 

(~T~) ^ ^^"^""^"^ and finally ?5i(0) = ^-^-^^^ € IR{'=»+i+fco)xi_ ^he proof of the 

theorem for i < m proceeds in two steps, 
(a) We first show that: 

U^{x) = nf (Aci-x) • F^ • i/i(0) where 
n-(x) = rM?(T)-n?(x-T)dr + D?(x). 

JO 

Consider the following system of inductively defined integral equations. Let Ci^x = 
Acj — X. 

U<'>\x) = 

ut^'^'Hx) = ^^MnT)■ut^'Hx+T)dT+nnc^,.)■Ft■uM■ 

Jo 



and 

ja,{0) 



K^'^'\ci,,) = Mf (r) ■ Uf^\ci,,-r) dr + D,"(c,,J. 

Jo 

Clearly, n«(ci,^) = lim WA^^^^a^x) and f/f (x) = lim U^'^^^^\x). 
By induction on j, we prove the following relation: 

Ut^^\x) = uf=\c,,,)-F'}.UM- 
(i) (Base case.) Ut^^\x) = and n"'^°^(ci,^) = 0. 



30 



T. CHEN, T. HAN, J.-P. KATOEN, AND A. MEREACRE 



(ii) (Induction step.) By exploiting the induction hypothesis (in the second step), 
we derive: 




Mf{T)Ut''>{x + T)dT + Dr(c,.) • F^Um 

(r)n^'(^')(c,,,-r) • F^Ui{0)dT + D^(ci,,) • F^Ui{0) 

M'}{T)nf^\ci,^ - T)dT + D^(c,,,) ) • FfC>i(0) 



(b) n-(Ac,) • F-uM = 

X = and we obtain 



i,x) ■ FiUi{Q). 

n,(Ac,)F,^,+i(o) + n°(AQ);7o(o) 
f/o(o) 



Let 



C/f(0)=n,«(c.,o)-F^C/,(0). 
We can also write the above relation for x = as: 



^7.(0) 

U'M 



n^^AcO ( I BJ; ) 



n,(Ac,) 












n^Ac.) ^ 


(4j 


I 



^7^+1(0) 
C/o(0) 



n,(Ac,)F, 









n^AQ) ^ 


V C^o(O) 



/ n,(Ac,)F,[7,+i(_o) + nf(Ac,)£/o(o) \ 

1, C/o(0) j ■ 

As a result we can represent [/j(0) in the following matrix form 
um = Ili{l^Ci)FiUi+m) + n,"(Aci)c7o(0) 

by noting that Ilj is formed by the first ki rows and columns of matrix 11" and 
is formed by the first ki rows and the last /c? — ki columns of 11?. 
(2) [i = m.) The proof of this case follows almost immediately from equation ()4.15p . As 
any region in Qm is unbounded, delay transitions do not exist. As Umix+r) does not 
depend on x, the integral 'M.m{T)Um{x+T) dr reduces to Mm(T) dr ■ Um{0). In 
addition, M.m{T) dr boils down to Pm and Bm(T) dr to B^- D 

Since the coefficients of the linear equations are all known, solving the system of linear 
equations yields f/o(0), which contains the probability Probvg{0) of reaching Vp from initial 
vertex vq. 

Theorem 14.91 is based on the equations (I4.14p (for i < m), and ()4.15p (for i=m). The 
term nj(Acj) -Fj • f7j+i(0) stands for the delay transitions, where Fj specifies how the delay 
transitions are connected between the sub-graphs Qi and Gi+i. The term n"(Acj) ■ Uo{0) 
stands for Markovian transitions with reset. The term II^(Acj) in the augmented CTMC Cf 
specifies the probabilities to first take transitions inside Gi followed by a one-step Markovian 
transition back to Gq. 
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Remark 4.10. The approach in this section is focused on single-cfock DTA (with finite 
acceptance criteria). For two-clock DTA^ the approach fails. In case of a single clock x, 
any reset (of x) from Qi yields a state in ^o(O), and any delay (of x) yields some state in 
^i+i(0). However, in the setting of two clocks, after a reset generally only one clock has a 
fixed value while the value of the other one is not determined. 

Lemma 4.11. For CTMC C and single-clock DTA^ A, computing Vv^ {^Paths^ {A.)^ can 
he done in time 0{m?-\S\-\Loc\-\-/S.c + m^-\Sf'-\Loc^), where m is the number of constants 
appearing in A, \S\ is the number of states in C, \Loc\ is the number of locations in A, A is 
the maximal exit rate in C and Ac = maxo^j<rrt{Aci}. 

Proof. The DMTA 00 A has at most |5|-|-Loc| locations. The number of vertices in the PDF 
Z{C0A) is at most m-|S'|-|Loc|, as there are m possible regions. CTMC Qi and its annotated 
version Qf thus have at most O {m-\S\-\Loc\) states. Calculating the transient distribution 
nj(Acj) on CTMC Qi for any state in Qi takes at most O {m-\S\-\Loc\-X-Ac) where A is the 
maximal exit rate in Qi (and thus in C) and Ac = maxo<gi<m{Aci} is the maximal width of 
a region. Given that this computation needs to be performed for any subgraph yields the 
first summand in the time complexity. Subsequently, according to Theorem 14.91 a system 
of linear equations has to be solved with at most O {m-\S\-\Loc\) variables. This takes at 
most O (m^-|5p-|Locp) operations. □ 



5. Verifying CTMCs Against Muller DTA Specifications 

Finally, we deal with the verification of CTMCs against DTA with Muller acceptance 
conditions. The procedure is very similar to the one for DTA with finite acceptance con- 
ditions. Let ^ be a DTA^, and C a CTMC. The region graph of the product C A is 
defined as before (cf. Def . 13.111 page [T8|) , except that the accepting set Vp is defined using 
bottom (or: terminal) SCCs (BSCCs for short). A strongly connected component (SCC) is 
terminal if it cannot be left once entered. 

Definition 5.1 (Region graph of DMTA'^). The region graph of DMTA"^ M = [Lac, X, 4, 
Locj^ E, — ^) is Q{M.) = (y, vq, Vf, A, ^), where V, vq, A and are defined as in Def. 13.111 
(page [T8]). and Vp = {v ^ B \ B £ aB^ where aB is the set of accepting BSCCs in Q{A4). 
BSCC B QV is accepting if there exists Lp € Locjr such that for any v B, vli G Lp. 



Example 5.2. Consider the DMTA"^ in Figure 6(c) with Locp = {Lp-^jLp^} with Lp^ = 
{ii,i2, ^3}) and Lp^ = {ii,i5,iQ}. Its region graph is depicted in Figure dH There is one 
accepting BSCC, whose vertices are colored gray, corresponding to the set Lp^. There is 
no BSCC corresponding to Lp-^, due to the presence of the sink vertices V12 and V14. These 
vertices are reachable from locations ii and £2 if x ^ 2. 

Two remarks are in order. A first observation is that the probability to stay in an 
accepting BSCC is one, considering both the delay and Markovian transitions. That is to 
say, there are no outgoing transitions from which some probability can "leak away". In 
addition, any pair of accepting BSCCs is disjoint, which allows the addition of, e.g., their 
reachability probabilities. 

Theorem 5.3. For any CTMC C, DTA'^ A, Fr'^ {Paths^ (A)) is the least solution of 
Probf^iO, U), where DTMP V = emh{Z{C (g) A)) and U = UseaB 
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Figure 11: Region graph of the product DMTA"^ in Figure 6(c 



Proof. To start off, observe that Fr'^ (^Paths^ (A)) is measurable, cf. Theorem 13.21 (page [TO j) . 
The proof follows from Theorem 13.101 and the following observations. For any DTMP 
expanded with a finite set of locations — like for finite DTMCs — almost surely the states 
that are visited infinitely often along a path constitute a BSCC. It thus follows that the 
probability for visiting a set of states infinitely often equals the reachability probability of 
some BSCC in the DTMP emb{Z{C (8) A)). The result now follows from Theorem 14.31 □ 

Example 5.4. Consider the region graph in Figure [TTl The only BSCC is indicated by the 
gray shaded states. To determine Ft'-' (^Paths'^ {A)) , it suffices to consider the reachability 

probability for T = {^1,^2}- For the delay transition vq ^ vg, we have 



Prob^oA^) = e-"«-i-Pro6„9(l) = e" 



ro-l 



•0 



0. 



OA,{x} 

For the Markovian transition vq ^ vi, 



Prob^^^y^{0) 



OA-ro-e-''°-^-Prob^,{T)dT 



0.6,{x} 



0.4To-e-'^«-^ dr. 



A similar reasoning applies to vq ^ V2. Gathering the results we obtain: 



Pr'^(Pai/i/(^)) = / (0.4 + 0.6)To-e-'o-^(ir = / 
^0 Jo 



ro-e 



'■^dr = 1 



-ro 



Verifying qualitative specifications. Until now we have investigated the quantitative 
verification problem, which is to determine the value of Pr(C \= A). The qualitative verifi- 
cation problem, on the other hand, is to determine whether the probability that C satisfies 
A exceeds zero, or, dually, equals one. For stochastic processes such as finite CTMCs and 
finite DTMCs, qualitative verification problems are known to be decidable by means of a 
simple graph analysis. 
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Proposition 5.5. For any CTMC C and DTA A, 

(1) Vi^[Paths^{A)) > iff Z{C®A) ^3<)Vf, 

(2) VT:^{Paths^{A)) =1 iff Z{C ® A) {{^<}Vf)^ Vp), 

where Vf={v^V \ v[i^Locf} for DTA^, Vf={v^B \ B€aB} for DTA", and W denotes 
the weak until operator. 

Proof. Similar to tlie case for discrete-time Markov chains [8, Chapter 10]. □ 

From the above theorem, it follows that the qualitative properties can be verified using 
a standard graph-based CTL model checking algorithm, i.e., by just considering the under- 
lying finite digraph of the PDF Z{C ^ A) — basically the region graph of C ® A — while 
ignoring the transition probabilities. 

6. Conclusion 

This paper addressed the quantitative (and qualitative) verification of a finite CTMC 
C against a linear real-time specification given as a deterministic timed automaton (DTA). 
We studied DTA with finite and Muller acceptance criteria. The key result (for finite 
acceptance) is that the probability of C ^ ^ equals the reachability probability in the 
embedded discrete-time Markov process of a PDF. This PDF is obtained via a standard 
region construction. Reachability probabilities in the thus obtained FDPs are characterized 
by a system of Volterra integral equations of the second type and are shown to be approx- 
imated by a system of FDEs. For Muller acceptance criteria, the probability of C ^ ^ 
equals the reachability probability of the accepting terminal SCCs in the embedded PDF. 
These results apply to DTA with arbitrarily (but finitely) many clocks. For single-clock 
DTA with finite acceptance, Fr(C |= A) is obtained by solving a system of linear equations 
whose coefficients are solutions of a system of ODEs. As the coefficients are in fact transient 
probabilities in CTMCs, this result implies that standard algorithms for CTMC analysis 
suffice to verify single-clock DTA specifications. 

An interesting future research direction is the verification against non-deterministic 
timed automata (NTA). NTA are strictly more expressive than DTA, and thus would allow 
more linear real-time specification. Following the approach in this paper requires a non- 
deterministic variant of PDF. Another challenging open problem is to consider real-time 
linear temporal logics as specifications such as metric temporal logic (MTL) [21] or variants 
thereof. 
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